Foundations  for  Meta-Analysis 


Developing  a  Taxonomy  of  Intelligence 
Analysis  Variables 

Rob  Johnston 


66 

Taxonomies  are 
signposts,  indicating 
what  is  known  and 
what  has  yet  to  be 
discovered. 

99 


Dr.  Rob  Johnston  is  a 

postdoctoral  research  fellow  at 
the  CIA  Center  for  the  Study  of 
Intelligence  and  a  member  of  the 
research  staff  at  the  Institute  for 
Defense  Analyses. 


Editor’s  Note:  By  distilling  a  list 
of  the  variables  that  affect  ana¬ 
lytic  reasoning,  the  author  aims 
to  move  the  tradecraft  of  intelli¬ 
gence  analysis  closer  to  a  science. 
A  carefully  prepared  taxonomy 
can  become  a  structure  for  height¬ 
ening  awareness  of  analytic 
biases,  sorting  available  data, 
identifying  information  gaps,  and 
stimulating  new  approaches  to 
the  understanding  of  unfolding 
events,  ultimately  increasing  the 
sophistication  of  analytic  judg¬ 
ments.  The  article  is  intended  to 
stimulate  debate  leading  to 
refinements  of  the  proposed  vari¬ 
ables  and  the  application  of  such 
a  framework  to  analytic  thinking 
among  intelligence  professionals. 


Science  is  organized  knowledge. 

Herbert  Spence1 

Aristotle  may  be  the  father  of 
scientific  classification,  but  it  was 
Carolus  Linnaeus  who  introduced 
the  first  formal  taxonomy — 
kingdom,  class,  order,  genera,  and 
species — in  his  Sy sterna  Naturae 
in  1735.  By  codifying  the  naming 
conventions  in  biology,  Linnaeus’s 
work  provided  a  reference  point 
for  future  discoveries.  Moreover, 
the  development  of  a  hierarchical 
grouping  of  related  organisms 

1  Herbert  Spencer  wrote  The  Study  of 
Sociology  in  1874,  which  set  the  stage 
for  sociology  to  emerge  as  a  discipline. 


contributed  significantly  to 
Darwin’s  creation  of  an 
evolutionary  theory.  The  Systema 
Naturae  taxonomy  was  not  a  fixed 
product,  but  rather  a  living 
document.  Linnaeus  himself 
revised  it  through  10  editions,  and 
later  biologists  have  continued  to 
modify  it.2 

As  discoveries  and  research 
methods  in  other  domains  grew, 
taxonomies  were  created  to  help 
organize  those  disciplines  and 
assist  researchers  in  identifying 
variables  that  required  addi¬ 
tional  study.  The  development  of 
specific  taxonomies— from  highly 
structured  systems  like  the  Peri¬ 
odic  Table  of  chemical  elements 
to  less  structured  approaches  like 
Bloom’s  Taxonomy3 — is  a  key 
step  in  organizing  knowledge  and 
furthering  the  growth  of  individ¬ 
ual  disciplines.  A  taxonomy 
differentiates  domains  by  bound¬ 
ing  the  problem  space,  codifying 
naming  conventions,  identifying 
areas  of  interest,  helping  to  set 

2  Ernst  Haeckel  introduced  phylum  to 
include  related  classes  and  family  to  in¬ 
clude  related  genera  in  1866.  The  Lin¬ 
naeus  taxonomy  is  currently  being 
revised  to  accommodate  genomic  map¬ 
ping  data. 

3  See  Benjamin  S.  Bloom,  ed.,  Taxonomy 
of  Educational  Objectives:  The  Classifi¬ 
cation  of  Educational  Goals:  Handbook 
I,  Cognitive  Domain  (New  York,  NY: 
Longmans,  1956).  Bloom's  taxonomy  is 
a  classification  of  levels  of  intellectual 
behavior  in  learning,  including  knowl¬ 
edge,  comprehension,  application,  anal¬ 
ysis,  synthesis,  and  evaluation. 
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Taxonomy 


research  priorities,  and  often 
leading  to  new  theories.  Taxono¬ 
mies  are  signposts,  indicating 
what  is  known  and  what  has  yet 
to  be  discovered. 

This  paper  proposes  a  taxonomy 
for  the  field  of  intelligence.  Over 
100  individuals  gave  their  time 
and  assistance  in  this  work.  The 
resulting  organized  listing  of 
variables  will  help  practitioners 
strengthen  their  understanding 
of  the  analytic  process  and  point 
them  in  directions  that  need 
additional  attention.4 


Intelligence  Analysis 

We  could  have  talked  about  the 
science  of  intelligence,  but .  .  . 
the  science  of  intelligence  is  yet 
to  be  invented. 

Charles  Allen5 


■*  1  would  like  to  thank  the  organizations 
that  facilitated  this  study,  specifically 
the  Central  Intelligence  Agency’s 
Center  for  the  Study  of  Intelligence;  the 
Defense  Intelligence  Agency;  the  Insti¬ 
tute  for  Defense  Analyses;  the  National 
Military  Intelligence  Association;  Evi¬ 
dence  Based  Research,  Incorporated; 
Analytic  Services,  Inc.  (ANSER);  and 
the  individual  analysts  at  other  organi¬ 
zations  who  gave  of  their  time.  Aminis- 
trators,  faculty  members,  and  students 
at  the  CIA’s  Kent  School,  the  Joint  Mili¬ 
tary  Intelligence  College,  the  Naval 
Postgraduate  School,  Columbia  Univer¬ 
sity,  Georgetown  University,  and  Yale 
University  also  supported  this  project. 

5  Comment  made  by  the  Associate 
Director  of  Central  Intelligence  for 
Collection  at  a  public  seminar  on  intelli¬ 
gence  at  Harvard  University,  spring 
2000.  See  <http://pirp.harvard.edu/pdf- 
blurb.asp?  id+518>. 
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Understanding  intelligence 
analysis  is  not  a  trivial  matter. 
The  literature  in  the  field  is 
episodic  and  reflects  specialized 
areas  of  concern.  Intelligence 
literature  makes  an  important 
distinction  between  solving  a 
problem  in  the  public  domain  and 
solving  a  problem  in  a  private  or 
secret  domain.  This  distinction 
seems  key  in  differentiating 
between  general  analysis  and 
intelligence  analysis. 

Ronald  Garst  articulates  two 
arguments  that  are  used  to  sup¬ 
port  this  distinction;  Intelligence 
analysis  is  more  time  sensitive 
than  analysis  in  other  domains, 
he  suggests,  and  it  deals  with 
information  that  intentionally 
may  be  deceptive.5  The  notion 
that  intelligence  is  uniquely  time 
sensitive  is  questionable.  Intelli¬ 
gence  is  not  the  only  domain 
where  time  constraints  can  force 
decisions  to  be  made  before  data 
are  complete.  Whether  one  is  in 
an  operating  room  or  a  cockpit, 
time  is  always  a  key  variable. 
Intelligence  is  a  life  and  death 


Ronald  Garst,  “Fundamentals  of 
Intelligence  Analysis,"  Ronald  Garst, 
ed.,  A  Handbook  of  Intelligence  Analy¬ 
sis,  2nd  ed.  (Washington,  DC:  Defense 
Intelligence  College,  1989). 


profession,  but  so  are  medicine 
and  mass  transportation.  In 
each  instance,  failure  can  mean 
casualties. 

Garst’s  point  about  intentional 
deception  is  more  germane, 
Seldom  do  analysts  in  fields 
other  than  intelligence  deal  with 
intentional  deception.  Michael 
Warner  makes  a  good  case 
for  secrecy  being  the  primary 
variable  that  distinguishes  intel¬ 
ligence  from  other  activities.1  He 
argues  that  the  behavior  of  the 
subject  of  intelligence  changes  if 
the  subject  is  aware  of  being 
observed  or  analyzed.  Not  only  is 
this  true  for  intentional  decep¬ 
tion,  the  argument  is  supported 
by  a  long  history  of  psychological 
research  beginning  with  an 
experimental  program  at  the 
Hawthorne  Plant  between  1927 
and  1930.  The  result  of  that 
research  was  the  theory  of  the 
“Hawthorne  Effect,”  which, 
broadly  interpreted,  states  that 
when  subjects  are  aware  of  being 
observed  their  behavior  changes.8 


7  Michael  Warner,  “Wanted:  A  Defini¬ 
tion  of  “Intelligence,”  Studies  in  Intelli¬ 
gence,  Vol.  46.  No.  3,  2002  fWashington, 
DC:  Center  for  the  Study  of  Intelli¬ 
gence,  2002). 

Fritz  J.  Roethlisberger  and  William  J. 
Dickson,  Management  and  the  Worker: 
An  Account  of  a  Research  Program  Con¬ 
ducted  by  the  Western  Electric  Company, 
Hawthorne  Works,  Chicago  (Boston, 
MA:  Harvard  University  Press,  1939); 
Elton  Mayo,  The  Human  Problems  of  an 
Industrial  Civilization  (New  York,  NY' 
MacMillan,  1933). 
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Intentional  deception  can  occur 
outside  intelligence — in 
connection  with  certain  law 
enforcement  functions,  for 
example — but  most  of  the 
professional  literature  treats  this 
as  the  exception,  rather  than  the 
rule.  In  the  case  of  intelligence 
analysis,  deception  is  the  rule.  In 
intelligence  analysis,  the  validity 
of  the  data  is  always  in  doubt. 
Moreover,  intelligence  analysts 
are  specifically  trained  to  factor 
in  deception  as  part  of  the 
analytic  process,  to  look  for 
anomalies  and  outliers  instead  of 
focusing  on  the  central 
tendencies  of  distribution. 

The  taxonomy  being  developed 
here  requires  a  definition  of 
intelligence  analysis  that  is 
specific  to  the  field.  Sherman 
Kent,  a  pioneer  in  the  intelli¬ 
gence  discipline,  wrote  that 
intelligence  was  a  “special 
category  of  knowledge.”9  He 
outlined  the  basic  descriptive 
element,  the  current  reporting 
element,  and  the  speculative 
estimates  element  as  the  key 
components  of  intelligence 
analysis.  His  work  laid  the 
foundation  for  understanding  the 
activities  inherent  in  intelligence 
analysis  by  demonstrating  that 
the  analytic  process  itself  was 
subject  to  being  analyzed.  Kent 
took  the  first  step  toward 


a  Sherman  Kent,  Strategic  Intelligence 
for  American  World  Policy  (Princeton. 
NJ:  Princeton  University  Press,  1966). 
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The  analytic  process  is 
a  construction  of  the 
human  mind  and  is 
significantly  different 
from  individual  to 
individual. 
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developing  a  higher  order,  or 
meta-analytic,  approach  to 
analysis  by  reducing  the  process 
to  smaller  functional  components 
for  individual  study. 

Following  suit,  other  authors 
focused  attention  on  the  process 
or  methodological  elements  of 
intelligence  analysis.  In  Intelli¬ 
gence  Research  Methodology, 
Jerome  Clauser  and  Sandra 
Weir  followed  Kent’s  three 
functional  areas  and  went  on 
to  describe  basic  research 
foundations  and  the  inductive 
and  deductive  models  for 
performing  intelligence 
analysis. 10  Garst’s  Handbook 
of  Intelligence  Analysis  contains 
less  background  in  basic 
research  methods  than  Clauser 
and  Weir,  but  it  is  more  focused 
on  the  intelligence  cycle.12 


10  Jerome  K.  Clauser  and  Sandra  M. 
Weir,  Intelligence  Research  Methodolo¬ 
gy:  An  Introduction  to  Techniques  and 
Procedures  for  Conducting  Research  in 
Defense  Intelligence  (Washington,  DC: 
US  Defense  Intelligence  School,  1976). 


Bruce  Berkowitz  and  Allan  Good¬ 
man  highlight  the  process  of 
strategic  intelligence  and  define 
intelligence  analysis  as:  “[T]he 
process  of  evaluating  and 
transforming  raw  data  into 
descriptions,  explanations,  and 
conclusions  for  intelligence 
consumers.”12  Lisa  Krizan,  too, 
focuses  on  process.  She  writes 
that,  “At  the  very  least,  analysis 
should  fully  describe  the  phe¬ 
nomenon  under  study,  accounting 
for  as  many  relevant  variables  as 
possible.  At  the  next  higher  level 
of  analysis,  a  thorough  expla¬ 
nation  of  the  phenomenon  is 
obtained,  through  interpreting 
the  significance  and  effects  of  its 
elements  on  the  whole.”13  In 
addition,  several  authors  have 
written  about  individual  analytic 
approaches,  including  the  LAMP 
method,  Warnings  of  Revolution, 


11  See  also:  Morgan  Jones,  The  Think¬ 
er's  Toolkit:  14  Powerful  Techniques  for 
Problem  Solving  (New  York,  NY:  Times 
Business,  1998).  Jones’s  book  is  a  popu¬ 
lar  version  of  both  Garst’s  and  Clauser 
and  Weir’s  work  in  that  it  describes  a 
collection  of  fourteen  analytic  methods 
and  techniques  for  problem-solving; 
however,  the  methods  are  not  necessari¬ 
ly  specific  to  intelligence. 

12  Bruce  D.  Berkowitz  and  Allan  E. 
Goodman,  Strategic  Intelligence  for 
American  National  Security  (Princeton, 
NJ:  Princeton  University  Press,  1989), 
p.  85. 

,J  Lisa  Knzan,  Intelligence  Essentials 
for  Everyone  (Washington,  DC:  Joint 
Military  Intelligence  College,  1999), 
p.  29. 
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Bayes’  Theorem,  Decision  Trees, 
and  FACTIONS  and  Policon.14 

Explicit  in  the  above  definitions 
is  the  view  that  analysis  is  both  a 
process  and  a  collection  of  spe¬ 
cific  techniques.  Analysis  is  seen 
as  an  action  that  incorporates  a 
variety  of  tools  to  solve  a  prob¬ 
lem.  Different  analytic  methods 
have  something  to  offer  different 
analytic  problems.  Although  the 
referenced  works  focus  on  meth¬ 
ods  and  techniques,  they  do  not 
suggest  that  analysis  is  limited 
to  tools  and  techniques. 

Implicit  in  the  above  definitions 
is  the  idea  that  analysis  is  a 
product  of  cognition.  Some 
authors  directly  link  analysis 
with  cognition.  Robert  Mathams 
defines  analysis  as:  “[T]he 
breaking  down  of  a  large  problem 
into  a  number  of  smaller 

14  See1.  Jonathan  Lockwood  and  K. 
Lockwood,  “The  Lockwood  Analytical 
Method  for  Prediction  (LAMP),”  Defense 
Intelligence  Journal,  3(2),  1994,  pp.  47- 
74;  R.  Hopkins,  Warnings  of  Revolution: 
A  Case  Study  of  El  Salvador  (Washing¬ 
ton,  DC;  Center  for  the  Study  of  Intelli¬ 
gence,  1980)  TR  80-100012;  Jack 
Zlotnick,  “Bayes'  Theorem  for  Intelli¬ 
gence  Analysis,"  (1972),  in  H.  Bradford 
Westerfield,  cd..  Inside.  CIA's  Private 
World:  Declassified  Articles  from  the 
Agency's  Internal  Journal  (New  Haven, 
CT:  Yale  University  Press,  1995);  John 
Pierce,  “Some  Mathematical  Methods 
for  Intelligence  Analysis,"  Studies  in 
Intelligence,  No.  21,  Summer,  1977, 
pp.  1-19  I  declassified);  Edwin  Sapp, 
“Decision  Trees, ’’  Studies  in  Intelligence, 
No  18,  Winter,  1974,  pp.  45-57  (declas¬ 
sified),  Stanley  Feder,  “FACTIONS  and 
Policon;  New  Ways  to  Analyze  Politics," 
(1987),  in  Westerfield,  ed.,  Inside  CIA’s 
Private  World. 


problems  and  performing  mental 
operations  on  the  data  in  order  to 
arrive  at  a  conclusion  or 
generalization.”15  Another 
scholar  writes;  “Since  the  facts 
do  not  speak  for  themselves  but 
need  to  be  interpreted,  it  is 
inevitable  that  the  individual 
human  propensities  of  an 
intelligence  officer  will  enter  into 
the  process  of  evaluation.”16  Yet 
others  describe  analysis  as  a 
process  whereby.  “[Ijnformation 
is  compared  and  collated  with 
other  data,  and  conclusions  that 
also  incorporate  the  memory  and 
judgment  of  the  intelligence 
analyst  are  derived  from  it.”17 


Several  authors  make  the  case 
that  analysis  is  not  just  a  prod¬ 
uct  of  cognition  but  is  itself  a 
cognitive  process.  J.  R.  Thomp¬ 
son  and  colleagues  write  that 
“[Ilntelligence  analysis  is  an 
internal,  concept-driven  activity 
rather  than  an  external  data- 
driven  activity.”18  In  his  Psychol- 


15  Robert  Mathams,  “The  Intelligence 
Analyst’s  Notebook,"  in  D.  Dearth  and 
R.  Goodden,  eds.,  Strategic  Intelligence: 
Theory  and  Application.  2nd  ed.,  (Wash¬ 
ington,  DC:  Joint  Military  Intelligence 
College,  1995),  p.  88. 
lfi  Avi  Shlaim,  “Failures  in  National  In¬ 
telligence  Estimates:  The  Case  of  the 
Yom  Kippur  War,”  World  Politics,  28(3), 
1976,  pp.  348-380. 

17  John  Quirk,  David  Phillips,  Ray 
Cline,  and  Walter  Pforzheimer,  The  Cen¬ 
tral  Intelligence  Agency:  A  Photograph¬ 
ic  History  (Guilford,  CT:  Foreign 
Intelligence  Press,  1986). 

14  J.  R.  Thompson,  R.  Hopf-Weichel,  and 
R.  Geiselman,  The  Cognitive  Bases  of  In¬ 
telligence  Analysis  (Alexandria,  VA: 
Army  Research  Institute,  Research  Re¬ 
port  1362,  1984),  AD-A146,  pp.  132,  7. 


ogy  of  Intelligence  Analysis, 
Richards  Heuer  observes:  “Intel¬ 
ligence  analysis  is  fundamentally 
a  mental  process,  but  under¬ 
standing  this  process  is  hindered 
by  the  lack  of  conscious  aware¬ 
ness  of  the  workings  of  our 
own  minds.”19  Ephraim  Kam 
comments:  “The  process  of  intel¬ 
ligence  analysis  and  assessment 
is  a  very  personal  one.  There  is 
no  agTeed-upon  analytical 
schema,  and  the  analyst  must 
primarily  use  his  belief  system  to 
make  assumptions  and  interpret 
information.  His  assumptions 
are  usually  implicit  rather  than 
explicit  and  may  not  be  apparent 
even  to  him.”20 

These  definitions  reflect  the 
other  end  of  the  spectrum  from 
those  concerned  with  tools  and 
techniques.  They  suggest  that 
the  analytic  process  is  a  constr¬ 
uction  of  the  human  mind  and  is 
significantly  different  from 
individual  to  individual  or  group 
to  group.  Certainly  Kam’s  view 
is  the  most  radical  departure,  but 
even  he  does  not  suggest  that  one 
forego  tools,  rather  that  the 
process  of  choosing  the  tool  is 
governed  by  cognition  as  well. 

Recognizing  that  the  scope  of 
intelligence  analysis  is  so  broad 
that  it  includes  not  only  methods 
but  also  the  cognitive  process  is 


19  Richards  J.  Heuer,  Jr.,  Psychology  of 
Intelligence  Analysis  (Washington,  DC: 
Center  for  the  Study  of  Intelligence, 
1999),  p.  1. 

20  Ephraim  Kam,  Surprise  Attack.  The 
Victim's  Perspective  (Cambridge,  MA: 
Harvard  University  Press,  1988),  p.  120. 
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a  significant  step.  Viewing  anal¬ 
ysis  as  a  cognitive  process  opens 
the  door  to  a  complex  array  of 
variables.  Not  only  does  one 
have  to  be  concerned  about  indi¬ 
vidual  analytic  tools,  but  one  also 
has  to  factor  in  the  psychology  of 
the  individual  analyst.  In  the 
broadest  sense,  this  means 
not  merely  understanding  the 
individual  psyche  but  also  under¬ 
standing  the  variables  that 
interact  with  that  psyche.  In 
other  words,  intelligence  analysis 
is  the  socio-cognitive  process  by 
which  a  collection  of  methods  is 
'•  used  to  reduce  a  complex  issue 

into  a  set  of  simpler  issues  within 
a  secret  domain. 


Developing  the  Taxonomy 

The  first  step  of  science  is  to 
know  one  thing  from  another. 
This  knowledge  consists  in 
their  specific  distinctions;  but 
in  order  that  it  may  be  fixed 
and  permanent  distinct  names 
must  be  given  to  different 
things,  and  those  names  must 
be  recorded  and  remembered. 

Carolus  Linnaeus, 
16th  century  biologist 

My  research  was  designed  to 
isolate  variables  that  affect  the 
analytic  process.  The  resulting 
taxonomy  is  meant  to  bound  the 
problem  space  and  stimulate 
dialogue  leading  to  refinements. 
Although  a  hierarchic  list  is 
artificial  and  rigid,  it  is  a  first 
step  in  clarifying  areas  for  future 
research.  The  actual  variables 
are  considerably  more  fluid  and 
interconnected  than  such  a 
structure  suggests.  They  might 
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eventually  be  better  represented 
by  a  link  or  web  diagram,  once 
the  individual  elements  are 
refined  through  challenges  in  the 
literature. 


To  create  this  intelligence  analy¬ 
sis  taxonomy,  I  took  Alexander 
Ervin’s  applied  anthropological 
approach,  which  uses  multiple 
data  collection  methods  to  trian¬ 
gulate  results.21  I  also  drew  on 
Robert  White's  mental  workload 
model,  David  Meister’s  behav¬ 
ioral  model,  and  the  cognitive 
process  model  by  Gary  Klein  and 
his  colleagues.22  Each  model 
focuses  on  a  different  aspect  of 
human  performance  for  the 
development  of  a  taxonomy: 
White’s  model  examines  the 


21  Alexander  Ervin,  Applied  Anthropolo¬ 
gy.  Tools  and  Perspectives  for  Contem¬ 
porary  Practice  (Boston,  MA:  Allyn  and 
Bacon,  2000). 

22  Robert  White,  Task  Analysis  Methods: 
Review  and  Development  of  Techniques 
for  Analyzing  Mental  Workload  in  Mul¬ 
tiple  Task  Situations  (St.  Louis,  MO: 
McDonnell  Douglas  Corporation,  1971), 
MDC  J5291;  David  Meister,  Behavioral 
Analysis  and  Measurement  Methods 
(New  York,  NY:  Wiley,  1985);  G.  Klein, 
G.  R.  Calderwood,  and  A.  Clinton-Ciroc- 
co.  Rapid  Decision  Making  on  the  Fire 
Ground  (Yellow  Springs,  OH:  Klein  As¬ 
sociates,  1985),  KA-TR-84-41-7  (pre¬ 
pared  under  contract  MDA903-85-G- 
0099  for  the  US  Army  Research  Insti¬ 
tute,  Alexandria,  VA). 


actual  task  and  task  require¬ 
ments;  Meister’s  looks  at  the 
behavior  of  individuals  perform¬ 
ing  a  task;  and  Klein’s  uses 
verbal  protocols  to  identify  the 
cognitive  processes  of  individu¬ 
als  performing  a  task. 

Surveying  the  literature.  My 
research  began  with  a  review  of 
the  literature  for  background 
information  and  the  identifica¬ 
tion  of  variables.  I  found  2,432 
case  studies,  journal  articles, 
technical  reports,  transcripts  of 
public  speeches,  and  books 
related  to  the  topic.  This  was 
then  narrowed  to  374  pertinent 
texts  on  which  a  taxonomy  of 
intelligence  analysis  could  be 
built.  These  texts  were  analyzed 
to  identify  individual  variables 
and  categories  of  variables  that 
affect  intelligence  analysis.23  The 
intelligence  literature,  produced 
by  academics  and  practitioners, 
tends  to  be  episodic  or  case- 
based.  This  is  not  unique  to  the 
field  of  intelligence.  A  number  of 
disciplines — medicine,  business, 
and  law,  for  example — are  also 
case-based  in  nature.  A  number 
of  the  texts  were  general  or  theo¬ 
retical  and  indicate  a  trend  in 
specialization  within  the  field. 
Again,  this  is  not  an  uncommon 
phenomenon. 

A  “Q-sort”  method  was  used  to 
analyze  the  texts.24  As  I  read 
each  text,  I  recorded  the 


23  A  copy  of  the  bibliography  and  search 
criteria  is  available  from  the  author. 

24  See  William  Stephenson,  The  Study  of 
Behavior :  Q-Technique  and  its  Method¬ 
ology  (Chicago.  IL:  University  of  Chica¬ 
go  Press,  1953). 
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variables  that  each  author 
identified.  These  variables  were 
then  sorted  by  similarity  into 
groups.  Four  broad  categories  of 
analytic  variables  emerged  from 
the  Q-sort  process.25 

Refining  the  prototype.  Next,  I 
used  the  preliminary  taxonomy 
derived  from  my  reading  of  the 
literature  to  structure  interviews 
with  51  substantive  experts  and 
39  intelligence  novices.  In 
tandem,  I  conducted  two  focus 
group  sessions,  with  five 
individuals  in  each  group.  The 
interviews  and  focus  group 
discussions  resulted  in  adding 
some  variables  to  each  category, 
moving  variables  between 
categories,  and  removing  some  in 
each  category  that  appeared 
redundant. 

Testing  in  a  controlled  setting. 
Finally,  to  compare  the  taxonomy 
with  specific  analytic  behaviors,  I 
watched  participants  in  a 
controlled  intelligence  analysis 
^training  environment.  Trainees 
were  given  information  on 
specific  cases  and  directed  to  use 
various  methods  to  analyze  the 
situations  and  generate  final 
products.  During  the  training 
exercises,  the  verbal  and  physical 
behavior  of  individuals  and 


25  I  would  like  to  credit  Dr.  Forrest 
Frank  of  the  Institute  for  Defense  Anal¬ 
yses  for  his  suggestions  regarding  the 
naming  convention  for  these  categories 
of  variables  as  they  appear  on  the  ac¬ 
companying  chart. 
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groups  were  observed  and 
compared  with  the  taxonomic 
model.  I  participated  in  a 
number  of  the  exercises  myself  to 
gain  a  better  perspective.  This 
process  corroborated  most  of  the 
recommendations  that  had  been 
made  by  the  experts  and  novices 
and  also  yielded  additional 
variables  for  two  of  the 
categories.26 

The  resulting  taxonomy  is  purely 
descriptive.  It  is  not  intended  to 
demonstrate  the  variance  or 
weight  of  each  variable  or 
category.  That  is,  the  listing  is 
not  sufficient  to  predict  the  effect 
of  any  one  variable  on  human 
performance.  The  intention  of 
the  enumeration  is  to  provide  a 
framework  for  aggregating 
existing  data  and  to  create  a 
foundation  for  future 
experimentation.  Once  the 
variables  are  identified  and 
previous  findings  have  been 
aggregated,  it  is  reasonable  to 
consider  experimental  methods 
that  would  isolate  and  control 
individual  variables  and,  in  time, 
indicate  sources  of  error  and 
potential  remediation. 


Systemic  Variables 

The  column  of  Systemic  Variables 
on  the  chart  incorporates  items 
that  affect  both  an  intelligence 
organization  and  the  analytic 
environment.  Organizational 
Variables  include  the  structure  of 
the  intelligence  organization,  the 
managerial/reporting  chain, 
workflow  diagrams,  leadership, 
management,  management 
practices,  the  working  culture, 
history  and  traditions,  social 
practices  within  the  organization, 
work  taboos,  and  organizational 
demographics.  They  also 
encompass  internal  politics,  the 
hierarchical  reporting  structure, 
and  material  and  human 


2,i  Throughout  the  project,  my  data  col¬ 
lection  method  consisted  of  written  field 
notes.  Anthropologists  traditionally 
include  specific  detail  from  participant 
input  or  direct  observation.  Usually  this 
is  in  the  form  of  precise  descriptions  of 
the  actual  behavior  of  participants  and 
transcripts  of  their  verbal  interactions. 

It  is  also  standard  practice  in  field  work 
to  capture  these  data,  and  the  data  from 
the  interviews  and  focus  groups,  on 
audio-  or  videotape.  These  practices 
were  not  followed  in  this  particular  case 
for  two  reasons:  First,  the  nature  of 
my  work  was  to  derive  categories  of  vari¬ 
ables  and  individual  variables  in  order  to 
create  a  taxonomy,  rather  than  to  docu¬ 
ment  actual  practices  and  procedures, 
and  using  the  prototype  taxonomy  to 
structure  the  interactions  was  sufficient 
for  this  purpose:  second,  the  nature  of 
intelligence  work,  the  environment  in 
which  it  occurs,  and  the  professional 
practitioners  require  that  certain  data  be 
restricted. 


66 


Taxonomy 


Taxonomy  of  Intelligence  Analysis  Variables 

Systemic  Variables 

Systematic  Variables 

Idiosyncratic  Variables 

Communicative  Variables 

Organization 

User  Requirements 

Weltanschauung  (Worldview) 

Formal 

Internal 

Operations 

Affiliation 

Inter-Organization 

Structure 

Information  Acquisition 

Familial 

Hierarchical 

Leadership 

Collection  Methods 

Cultural 

Inter-Division 

Culture 

Overt 

Ethnic 

Inter-Group 

History 

Covert 

Religious 

Intra-Organization 

T  radition 

Information  Reliability 

Social 

Hierarchical 

Social  Practice 

Reproducible 

Linguistic 

Intra-Division 

Taboo 

Consistent 

Political 

Intra-Group 

Group  Characteristics 

Information  Validity 

Psychology 

Individual 

Hierarchy 

Historical 

Bias 

Hierarchical 

Resources  &  Incentives 

Single  Source 

Personality  Profile 

Inter-Group 

Manpower 

Dual  Source 

Security/T  rust 

Intra-Group 

Budget 

Triangulation 

Cognitive  Processing 

Informal 

Technology 

Information  Archive 

Learning  Style 

Inter-Organization 

Assets 

Storage 

Information  Acquisition 

Hierarchical 

R&D 

Access 

Information  Processing 

Inter-Division 

Facilities 

Correlation 

Expertise 

Inter-Group 

Work  Groups-Teams 

Retrieval 

Problem-Solving 

Intra-Organization 

External 

Analytic  Methodology 

Decisionmaking 

Hierarchical 

Consumer  Needs 

Approach 

Cognitive  Load 

Intra-Division 

Time  and  Imperatives 

Intuitive 

Speed/Accuracy 

Intra-Group 

Consumer  Use 

Structured 

Stress  Effects 

Individual 

Consumer  Structure 

Semi-Structured 

Education 

Hierarchical 

Consumer  Hierarchy 

Information  Processing 

Domain 

Inter-Group 

Consumer  Reporting 

Historical  Information 

Location 

Intra -Group 

Politics 

Current  Information 

Mentor 

Technology 

Internal  -  Organization 

Decision  Strategies 

Training 

Networked  Analysis 

Policy 

Tradition 

Taboo 

Security/Access 

External  -  National 

Law 

Policy 

External  -  international 
Security 

Denial 

Deception 

Policy 

Estimative 

Predictive 

Reporting 

Verbal  Methods 

Written  Methods 

Organizational 

Domain 

Procedural 

Readiness 

Resources 

Facilities 

Collaboration 
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resources.  The  fields  of 
industrial  and  organizational 
psychology,  sociology,  and 
management  studies  in  business 
have  brought  attention  to  the 
importance  of  organizational 
behavior  and  the  effect  it  has  on 
individual  work  habits  and 
practices.  Within  the  field  of 
intelligence,  the  works  of  Allison, 
Berkowitz  and  Goodman,  Elkins, 
Ford,  Godson,  and  Richelson, 
among  others,  examine  in 
general  the  organizational 
aspects  of  intelligence.27 

The  Systemic  Variables  category 
also  focuses  on  environmental 
variables — that  is,  external  influ¬ 
ences  on  the  organization,  such 
as  consumer  needs  and  require¬ 
ments,  time  constraints,  the 
consumer’s  model  for  using  the 
information,  the  consumer’s  orga¬ 
nization,  political  constraints, 
and  security  issues.  The  works  of 
Betts,  Hulnick,  Hunt,  Kam,  and 


27  Graham  T.  Allison,  Essence  of  Deci¬ 
sion:  Explaining  the  Cuban  Missile  Cri¬ 
sis  (Boston,  MA:  Little,  Brown  and 
Company,  1971);  Bruce  D.  Berkowitz 
and  Allan  E.  Goodman,  Best  Truth.  In¬ 
telligence  in  the  Information  Age  (New 
Haven,  CT:  Yale  University  Press, 
2000};  Dan  Elkins.  An  Intelligence  Re¬ 
source  Manager's  Guide  (Washington, 
DC:  Joint  Military  Intelligence  College, 
1997);  Harold  Ford,  Estimative  Intelli¬ 
gence:  The  Purposes  and  Problems  of 
National  Intelligence  Estimating  (Lan- 
ham.MD:  University  Press  of  America, 
1993};  Roy  Godson,  ed..  Comparing  For¬ 
eign  Intelligence:  The  U.S.,  the  USSR, 
the  U.K.  and  the  Third  World  (Washing¬ 
ton,  DC;  Pergamon-Brassey,  1988);  Jef¬ 
frey  Richelson,  The  U.S.  Intelligence 
Community,  4th  ed.  (Boulder,  CO:  West- 
view  Press,  1999). 


44 

The  taxonomy  is 
purely  descriptive.  It 
is  not  intended  to 
demonstrate  the 
weight  of  each  variable 
or  category. 
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Laqueur  address  the  environ¬ 
mental  and  consumer  issues  that 
affect  intelligence  analysis.28 

Case  studies  that  touch  on  differ¬ 
ent  Systemic  Variables  include; 
Allison,  on  the  Cuban  missile 
crisis;  Betts,  on  surprise  attacks; 
Kirkpatrick,  on  World  War  II 
tactical  intelligence  operations; 
Shiels,  on  government  failures; 
Wirtz,  on  the  Tet  offensive  in 
Vietnam;  and  Wohlstetter,  on 
Pearl  Harbor. 29 


Systematic  Variables 

The  Systematic  Variables  are 
those  that  affect  the  process  of 
analysis  itself.  They  include  the 
user’s  specific  requirements,  how 
the  information  was  acquired. 


28  Richard  K.  Betts,  “Policy-makers  and 
Intelligence  Analysts:  Love.  Hate  or 
Indifference,’'  Intelligence  and  National 
Security,  3(1),  January  1988,  pp.  184- 
189;  Arthur  S.  Hulnick,  “The  Intelli¬ 
gence  Producer-Policy  Consumer 
Linkage:  A  Theoretical  Approach," 
Intelligence  and  National  Security,  1(2), 
May  1986,  pp.  212-233;  David  Hunt, 
Complexity  and  Planning  in  the  21st 
Century:  Intelligence  Requirements  to 
Unlock  the  Mystery  (Newport,  RI:  Naval 
War  College,  2000);  Kam,  Surprise  At¬ 
tack: ;  Walter  A  Laqueur,  The  Uses  and 
Limits  of  Intelligence  (New  Brunswick, 
NJ;  Transaction  Publishers,  1993). 


the  information’s  reliability  and 
validity,  how  the  information  is 
stored ,  the  prescribed  methods 
for  analyzing  and  processing  the 
information,  specific  strategies 
for  making  decisions  about  the 
information,  and  the  methods 
used  to  report  the  information  to 
consumers. 

A  number  of  authors  have  writ¬ 
ten  about  the  analytic  tools  and 
techniques  used  in  intelligence: 
Clauser  and  Weir,  on  intelligence 
research  methods;  Jones,  on 
analytic  techniques;  and  Heuer, 
on  alternative  competing  hypoth¬ 
eses,  to  name  a  few.  Little  work 
has  been  done  comparing  struc¬ 
tured  techniques  to  intuition. 
Folker's  work  is  one  of  the  excep¬ 
tions — it  compares  the 
effectiveness  of  a  modified  form 
of  alternative  competing  hypothe¬ 
ses  with  intuition  in  a  controlled 
experimental  design.30  His  study 
is  unique  in  the  field  and  demon¬ 
strates  that  experimental 
methods  are  possible.  Krotow’s 
research,  on  the  other  hand, 
looks  at  differing  forms  of  cogni¬ 
tive  feedback  during  the  analytic 


28  Allison,  Essence  of  Decision;  Richard 
K.  Betts,  Surprise  Attack  (Washington, 
DC:  The  Brookings  Institution,  1982); 
Lyman  B.  Kirkpatrick,  Jr.,  Captains 
Without  Eyes:  Intelligence  Failures  in 
World  War  II  (London;  MacMillan 
Company,  1969);  Frederick  L.  Shiels, 
Preventable  Disasters:  iVTiy  Govern¬ 
ments  Fail  (Savage,  MD:  Rowman  and 
Littlefield  Publishers,  1991);  James  J. 
Wirtz,  The  Tet  Offensive:  Intelligence 
Failure  in  War  (Ithaca,  NY:  Cornell 
University  Press,  1991),  Roberta 
Wohlstetter.  Pearl  Harbor:  Warning 
and  Decision  (Palo  Alto,  CA:  Stanford 
University  Press,  1962). 
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process  and  makes  recommenda¬ 
tions  to  enhance  intelligence 
decisionmaking. 37 

Idiosyncratic  Variables 

Variables  in  the  third  column 
on  the  chart  are  those  that 
impact  on  an  individual  and 
his  or  her  analytic  performance. 
They  affect  the  individual’s 
Weltanschauung.  Although  the 
meaning  of  the  German  term  is 
difficult  to  capture  in  English, 
Sigmund  Freud  comes  close: 
“[A]n  intellectual  construction 
which  gives  a  unified  solution  of 
all  the  problems  of  our  existence 
in  virtue  of  a  comprehensive 
hypothesis,  a  construction, 
therefore,  in  which  no  question 
is  left  open  and  in  which  every¬ 
thing  in  which  we  are  interested 
finds  a  place.”32  Weltanscha¬ 
uung  has  been  translated  as 
“world  view,”  “mindset,”  and 
“mental  model,”  but  the  best 
approximation  in  English,  in  my 


30  MSgt.  Robert  D.  Folker,  Intelligence 
Analysis  in  Theater  Joint  Intelligence 
Centers:  An  Experiment  in  Applying 
Structured  Methods  (Washington,  DC: 
Joint  Military  Intelligence  College, 
2000),  Occasional  Paper  #7.  Folker’s 
study  has  some  methodological  flaws. 
Specifically,  it  does  not  describe  one  of 
the  independent  variables  (intuitive 
method),  leaving  the  dependent 
variable  (test  scores)  in  doubt. 

31  Geraldine  Krotow,  The  Impact  of 
Cognitive  Feedback  on  the  Performance 
of  Intelligence  Analysts  (Monterey,  CA: 
Nava!  Postgraduate  School,  1992), 
AD-A252,  p,  176. 

32  Sigmund  Freud,  “A  Philosophy  of  Life: 
Lecture  35,”  New  Introductory  Lectures 
on  Psycho-analysis  (London:  Hogarth 
Press,  1933). 


view,  is  the  often-overused  word 
“paradigm.”  Paradigm  stands 
for  the  sum  of  life’s  experiences 
and  acculturation  that  identi¬ 
fies  an  individual  as  a  member 
of  a  group.  In  the  proposed  tax¬ 
onomy,  Idiosyncratic  Variables 
include  one’s  familial,  cultural, 
ethnic,  religious,  linguistic,  and 
political  affiliations.  They  also 
encompass  psychological  factors 
like  biases,  personality  profiles, 
cognitive  styles  and  processing, 
cognitive  loads,33  expertise, 
approach  to  problem-solving, 
decisionmaking  style,  and  reac¬ 
tion  to  stress.  Finally,  there  are 
domain  variables  like  educa¬ 
tion,  training,  and  the  readiness 
to  apply  knowledge,  skills,  and 
abilities  to  the  task  at  hand. 

The  relevant  psychological  litera¬ 
ture  is  robust.  Amos  Tversky 
and  Daniel  Kahneman  began  to 
examine  psychological  biases  in 
the  early  1970s.34  Their  work 
has  found  its  way  into  the  intelli¬ 
gence  literature  through  authors 
like  Alexander  Butterfield,  Jack 
Davis,  James  Goldgeier,  and 
Richards  Heuer.36  Decisionmak¬ 
ing  and  problem-solving  have 
been  studied  since  the  early 
1920s  and  these  data  are 
reflected  in  Heuer’s  work  as 


ss  “Cognitive  loads"  are  the  amount/ 
number  of  cognitive  tasks  weighed 
against  available  cognitive  processing 
power. 

34  Amos  Tversky  and  Daniel  Kahne¬ 
man,  "The  Belief  in  the  'Law  of  Small 
Numbers,’”  Psychological  Bulletin ,  76, 
1971,  pp.  105-110;  Tversky  and  Kahne¬ 
man,  “Judgment  Under  Uncertainty: 
Heuristics  and  Biases,”  Science,  185, 
1974,  pp.  1124-1131. 


well,36  Personality  profiling,  too, 
is  well  understood  and  has  had 
an  impact  on  recent  intelligence 
practices  and  theory.37 

Other  well-researched  areas, 
however,  have  yet  to  be  studied 
in  the  context  of  intelligence. 
Issues  of  acculturation  and 
affiliation,  educational  factors, 
and  training  strategies,  for 
example,  may  yet  yield 
interesting  results  and  insights 
into  the  field  of  intelligence. 

Communicative  Variables 

The  fourth  category  contains 
variables  that  affect  interaction 
within  and  between  groups. 
Because  communication  is  the 
vital  link  within  the  system — 
among  processes  and  individu¬ 
als — this  group  of  variables  could 
be  included  logically  in  each  of 
the  other  three  categories.  Its 
broad  relevance,  however,  makes 
it  seem  reasonable  to  isolate  it  as 


35  Alexander  Butterfield,  The  Accuracy 
of  Intelligence  Assessment:  Bias ,  Percep¬ 
tion,  and  Judgment  in  Analysis  and  De¬ 
cision  (Newport,  RI:  Naval  War  College, 
1993),  AD-A266,  p.  925;  Jack  Davis, 
“Combating  Mindset,"  Studies  m  Intelli¬ 
gence,  Vol.  35,  Winter,  1991,  pp.  13-18; 
James  M.  Goldgeier,  “Psychology  and 
Security,”  Security  Studies,  6(4),  1997, 
pp.  137-166;  Heuer,  Psychology  of  Intel¬ 
ligence  Analysis. 

3li  Frank  H.  Knight,  Risk,  Uncertainty 
and  Profit  (Boston,  MA:  Houghton  Mif¬ 
flin,  1921). 

37  Caroline  Ziemke,  Philippe  Lous- 
taunau,  and  Amy  Alrich,  Strategic  Per¬ 
sonality  and  the  Effectiveness  of  Nuclear 
Deterrence  (Washington,  DC:  Institute 
for  Defense  Analyses,  2000),  D-2537. 
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a  distinct  area  of  variability.  The 
Communicative  Variables  include 
formal  and  informal  communica¬ 
tions  within  an  organization 
(from  products  to  e-mails);  among 
organizations;  and  between  indi¬ 
viduals  and  the  social  networks 
that  they  create.  In  his  essay  on 
estimative  probability,  Sherman 
Kent  highlights  this  area  by 
describing  the  difficulty  that 
producers  of  intelligence  have  in 
communicating  the  likelihood  of 
an  event  to  consumers  of  intelli¬ 
gence.38  Case  studies  by 
Wohlstetter  and  others,  which 
have  addressed  organizational 
issues,  also  touch  on  communica¬ 
tion  and  social  networks  and  the 
impact  that  communication  has 
on  the  analytic  process.39  This  is 
an  area  that  could  benefit  from 
additional  study. 

Conclusion 

There  is  rarely  any  doubt  that 
the  unconscious  reasons  for 
practicing  a  custom  or  sharing 
a  belief  are  remote  from  the 
reasons  given  to  justify  them. 

Claude  Levi-Strauss40 


38  Sherman  Kent,  “Words  of  Estimative 
Probability,"  Sherman  Kent  and  the 
Board  of  National  Estimates:  Collected 
Essays  (Washington,  DC:  Center  for  the 
Study  of  Intelligence,  1994). 

39  Wohlstetter,  Pearl  Harbor. 

40  Claude  Levi-Strauss  wrote 
Structural  Anthropology  in  1958, 
setting  the  stage  for  structuralism  to 
emerge  as  an  analytic  interpretive 
method. 
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small  task,  but  it  is  one 
that  is  required  for  the 
health  of  the 
profession. 
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Intelligence  analysis  is  art  and 
tradecraft.  There  are  specific 
tools  and  techniques  to  help  per¬ 
form  the  tasks,  but,  in  the  end,  it 
is  left  to  individuals  to  use  their 
best  judgment  in  making  deci¬ 
sions.  This  is  not  to  say  that 
science  is  not  a  part  of  intelli¬ 
gence  analysis.  Science  is  born 
of  organized  knowledge,  and 
organizing  knowledge  requires 
effort  and  time.  The  work  on  this 
taxonomy  is  intended  to  help  that 
process  by  sparking  discussion, 
identifying  areas  where  research 
exists  and  ought  to  be  incorpo¬ 
rated  into  the  organizational 
knowledge  of  intelligence,  and 
identifying  areas  where  not 
enough  research  has  been 
performed. 

The  field  of  medicine  has  a 
number  of  parallels.  Like 
intelligence,  the  practice  of 
medicine  is  an  art  and  a 
tradecraft.  Practitioners  are 
trusted  to  use  their  best  judgment 
in  problem-solving  by  drawing  on 
their  expertise.  What  is 
important  to  remember  is  that 
there  are  numerous  basic  sciences 
driving  medical  practice.  Biology, 
chemistry,  physics,  and  all  of  the 
subspecialties  blend  together  to 
create  the  medical  sciences,  the 
foundation  on  which  the  practice 
of  modem  medicine  rests.  The 


practice  of  medicine  has  been 
revolutionized  by  the  sciences 
that  underpin  its  workings. 

Intelligence  analysis  has  not 
experienced  that  revolution.  The 
basic  sciences  that  underpin 
intelligence  are  not  physical 
sciences.  It  is  difficult  to 
measure  what  is  meant  by 
“progress”  in  the  human  sciences. 
The  human  sciences  are 
considerably  more  multivariate 
than  the  physical  sciences  and  it 
is  much  more  difficult  to  control 
those  variables. 

There  are  numerous  domains  from 
which  intelligence  may  borrow. 
Organizational  behavior  is  better 
understood  today  than  ever  before. 
Problem-solving  and  decision¬ 
making  have  been  researched 
since  the  1920s.  Structural 
anthropology  addresses  many  of 
the  acculturation  and  identity 
issues  that  affect  individual 
behavior.  Cognitive  scientists  are 
building  models  that  can  be  tested 
in  experimental  conditions  and 
used  for  developing  new  tools  and 
techniques.  Sociology  and  social 
theory  have  much  to  offer  in 
studying  social  networks  and 
communication. 

The  organization  of  knowledge  in 
medicine  is  medical  science.  It 
took- thousands  of  years  to  turn 
folk  medicine  into  research-based 
allopathic  medicine.  The  cases 
that  are  used  to  build  expertise, 
the  tools  and  techniques  that 
support  diagnosis  and  treatment, 
and  the  criteria  on  which 
judgments  are  made  all  have 
integrated  the  art  of  medicine 
with  the  science  of  medicine. 
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The  organization  of  knowledge  in 
intelligence  is  not  a  small  task, 
but  it  is  one  that  is  required  for 
the  health  of  the  profession.  The 
taxonomy  proposed  here  could 
serve  as  a  springboard  for  a  num¬ 
ber  of  innovative  projects: 
development  of  a  research  matrix 
that  identifies  what  is  known  anJ 
how  that  information  may  be  of 
use  in  intelligence  analysis;  set¬ 
ting  a  research  agenda  in  areas 
of  intelligence  that  have  been 


insufficiently  studied;  applica¬ 
tion  of  research  from  other 
domains  to  develop  additional 
training  and  education  programs 
for  analysts;  creation  of  a  data¬ 
base  of  lessons  learned  and  best 
practices  to  build  a  foundation 
for  an  electronic  performance 
support  system;  integration  of 
those  findings  into  new  analytic 
tools  and  techniques;  and 
development  of  a  networked 
architecture  for  collaborative 


problem-solving  and  forecasting, 
to  name  a  few.  It  is  my  hope  that 
this  taxonomy  will  help  intelli¬ 
gence  practitioners  take  steps  in 
some  of  these  new  directions. 
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